{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "2.Bidirectional_RNN_IMDB_Numeric.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8hwAV-B64Evk",
        "colab_type": "text"
      },
      "source": [
        "#  Training and evaluating an LSTM using reversed sequences\n",
        " All you need to\n",
        "do is write a variant of the data generator where the input sequences are reverted along\n",
        "the time dimension \n",
        "\n",
        "(replace the last line with yield samples[:, ::-1, :], targets)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Mt7Ihd6i3Mjn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.datasets import imdb\n",
        "from tensorflow.keras.preprocessing import sequence\n",
        "from tensorflow.keras import layers\n",
        "from tensorflow.keras.models import Sequential"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jcNsJnta4LUc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "max_features = 10000\n",
        "maxlen = 500\n",
        "\n",
        "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y5y5Q74v4SGs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Reverses sequences \n",
        "Rx_train = [x[::-1] for x in x_train]\n",
        "Rx_test = [x[::-1] for x in x_test]"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W0iedJpNgzhI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# padding to enshoure all the sentence are of same length\n",
        "\n",
        "Rx_train = sequence.pad_sequences(x_train, maxlen=maxlen)\n",
        "Rx_test = sequence.pad_sequences(x_test, maxlen=maxlen)"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OZBIaztc6Bes",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# padding to enshoure all the sentence are of same length\n",
        "# NORMAL SEQUENCE\n",
        "\n",
        "x_train = sequence.pad_sequences(x_train, maxlen=maxlen)\n",
        "x_test = sequence.pad_sequences(x_test, maxlen=maxlen)"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cquTtSXwec6E",
        "colab_type": "text"
      },
      "source": [
        "# LSTM Using Reverse Seqence"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MvwSy4Cl6E07",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 260
        },
        "outputId": "4b68faec-f82c-41c2-e07e-bcad042c7a75"
      },
      "source": [
        "model = Sequential()\n",
        "model.add(layers.Embedding(max_features, 128))\n",
        "model.add(layers.LSTM(32))\n",
        "model.add(layers.Dense(1, activation='sigmoid'))\n",
        "\n",
        "model.compile(optimizer='rmsprop',\n",
        "loss='binary_crossentropy',\n",
        "metrics=['acc'])\n",
        "\n",
        "model.summary()"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding (Embedding)        (None, None, 128)         1280000   \n",
            "_________________________________________________________________\n",
            "lstm (LSTM)                  (None, 32)                20608     \n",
            "_________________________________________________________________\n",
            "dense (Dense)                (None, 1)                 33        \n",
            "=================================================================\n",
            "Total params: 1,300,641\n",
            "Trainable params: 1,300,641\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M1rmqymw6N8x",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 364
        },
        "outputId": "761ba3de-bcd6-4c7b-96f5-b5bf57b4bed3"
      },
      "source": [
        "history = model.fit(Rx_train, y_train,\n",
        "                    epochs=10,\n",
        "                    batch_size=128,\n",
        "                    validation_split=0.2)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "157/157 [==============================] - 8s 49ms/step - loss: 0.4474 - acc: 0.7940 - val_loss: 0.3147 - val_acc: 0.8750\n",
            "Epoch 2/10\n",
            "157/157 [==============================] - 7s 45ms/step - loss: 0.2749 - acc: 0.8956 - val_loss: 0.3786 - val_acc: 0.8266\n",
            "Epoch 3/10\n",
            "157/157 [==============================] - 7s 45ms/step - loss: 0.2269 - acc: 0.9139 - val_loss: 0.2800 - val_acc: 0.8912\n",
            "Epoch 4/10\n",
            "157/157 [==============================] - 7s 45ms/step - loss: 0.1980 - acc: 0.9277 - val_loss: 0.3133 - val_acc: 0.8628\n",
            "Epoch 5/10\n",
            "157/157 [==============================] - 7s 45ms/step - loss: 0.1675 - acc: 0.9395 - val_loss: 0.3640 - val_acc: 0.8668\n",
            "Epoch 6/10\n",
            "157/157 [==============================] - 7s 45ms/step - loss: 0.1497 - acc: 0.9484 - val_loss: 0.3679 - val_acc: 0.8776\n",
            "Epoch 7/10\n",
            "157/157 [==============================] - 7s 45ms/step - loss: 0.1373 - acc: 0.9522 - val_loss: 0.3467 - val_acc: 0.8844\n",
            "Epoch 8/10\n",
            "157/157 [==============================] - 7s 46ms/step - loss: 0.1242 - acc: 0.9582 - val_loss: 0.3948 - val_acc: 0.8654\n",
            "Epoch 9/10\n",
            "157/157 [==============================] - 7s 46ms/step - loss: 0.1095 - acc: 0.9622 - val_loss: 0.3801 - val_acc: 0.8782\n",
            "Epoch 10/10\n",
            "157/157 [==============================] - 7s 45ms/step - loss: 0.1015 - acc: 0.9643 - val_loss: 0.3686 - val_acc: 0.8784\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tfpfMPcmf3p4",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        },
        "outputId": "ac7c6088-abdb-4b0e-b128-6c71e11ccd8a"
      },
      "source": [
        "evaluation = model.evaluate(Rx_test,y_test,verbose=2)\n",
        "print()\n",
        "print(\"Loss: \",evaluation[0]*100,\"%\")\n",
        "print(\"Accuracy: \",evaluation[1]*100,\"%\")"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "782/782 - 8s - loss: 0.4058 - acc: 0.8652\n",
            "\n",
            "Loss:  40.58133661746979 %\n",
            "Accuracy:  86.52399778366089 %\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0Tg9D2l57sL_",
        "colab_type": "text"
      },
      "source": [
        "You get performance nearly identical to that of the chronological-order LSTM.\n",
        "Remarkably, on such a text dataset, reversed-order processing works just as well as\n",
        "chronological processing, confirming the hypothesis that, although word order does\n",
        "matter in understanding language"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xK0efR_6enZ8",
        "colab_type": "text"
      },
      "source": [
        " # Bidirectional_RNN By processing a sequence both ways, a bidirectional RNN can catch patterns that may be overlooked by a unidirectional RNN."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QQ49Jpcu8Hib",
        "colab_type": "text"
      },
      "source": [
        "#  Training and evaluating a bidirectional LSTM"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wbjezsfp89pc",
        "colab_type": "text"
      },
      "source": [
        "To instantiate a bidirectional RNN in Keras, you use the Bidirectional layer, which takes\n",
        "as its first argument a recurrent layer instance. Bidirectional creates a second, separate\n",
        "instance of this recurrent layer and uses one instance for processing the input sequences\n",
        "in chronological order and the other instance for processing the input sequences in\n",
        "reversed order."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I_YEkZHY6Q3v",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = Sequential()\n",
        "model.add(layers.Embedding(max_features, 32))\n",
        "model.add(layers.Bidirectional(layers.LSTM(32)))\n",
        "model.add(layers.Dense(1, activation='sigmoid'))\n",
        "\n",
        "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-NwNhn4V8Q4M",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 364
        },
        "outputId": "456070ed-2044-44d4-bd3e-42415565e85f"
      },
      "source": [
        "history = model.fit(x_train, y_train,\n",
        "                    epochs=10,\n",
        "                    batch_size=128,\n",
        "                    validation_split=0.2)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "157/157 [==============================] - 8s 51ms/step - loss: 0.5123 - acc: 0.7510 - val_loss: 0.3418 - val_acc: 0.8634\n",
            "Epoch 2/10\n",
            "157/157 [==============================] - 7s 47ms/step - loss: 0.3020 - acc: 0.8809 - val_loss: 0.3726 - val_acc: 0.8378\n",
            "Epoch 3/10\n",
            "157/157 [==============================] - 7s 46ms/step - loss: 0.2378 - acc: 0.9129 - val_loss: 0.2870 - val_acc: 0.8846\n",
            "Epoch 4/10\n",
            "157/157 [==============================] - 7s 47ms/step - loss: 0.2034 - acc: 0.9241 - val_loss: 0.3349 - val_acc: 0.8732\n",
            "Epoch 5/10\n",
            "157/157 [==============================] - 7s 46ms/step - loss: 0.1795 - acc: 0.9364 - val_loss: 0.4044 - val_acc: 0.8582\n",
            "Epoch 6/10\n",
            "157/157 [==============================] - 7s 47ms/step - loss: 0.1592 - acc: 0.9417 - val_loss: 0.3965 - val_acc: 0.8826\n",
            "Epoch 7/10\n",
            "157/157 [==============================] - 7s 47ms/step - loss: 0.1445 - acc: 0.9485 - val_loss: 0.3876 - val_acc: 0.8854\n",
            "Epoch 8/10\n",
            "157/157 [==============================] - 7s 46ms/step - loss: 0.1313 - acc: 0.9542 - val_loss: 0.3075 - val_acc: 0.8830\n",
            "Epoch 9/10\n",
            "157/157 [==============================] - 7s 47ms/step - loss: 0.1211 - acc: 0.9568 - val_loss: 0.4582 - val_acc: 0.8404\n",
            "Epoch 10/10\n",
            "157/157 [==============================] - 7s 46ms/step - loss: 0.1100 - acc: 0.9625 - val_loss: 0.3754 - val_acc: 0.8758\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Tke6KG0cgPbw",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        },
        "outputId": "d2b29cc9-a1d1-47a8-e5f1-ae3a4cc0bb81"
      },
      "source": [
        "evaluation = model.evaluate(x_test,y_test,verbose=2)\n",
        "print()\n",
        "print(\"Loss: \",evaluation[0]*100,\"%\")\n",
        "print(\"Accuracy: \",evaluation[1]*100,\"%\")"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "782/782 - 12s - loss: 0.4088 - acc: 0.8656\n",
            "\n",
            "Loss:  40.87960422039032 %\n",
            "Accuracy:  86.56399846076965 %\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F5RaxIP19rUD",
        "colab_type": "text"
      },
      "source": [
        "It performs slightly better than the regular LSTM you tried in the previous section,\n",
        "achieving over 89% validation accuracy. It also seems to overfit more quickly, which is\n",
        "unsurprising because a bidirectional layer has twice as many parameters as a chronological LSTM. With some regularization, the bidirectional approach would likely be a\n",
        "strong performer on this task."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UeookLKT9t4T",
        "colab_type": "text"
      },
      "source": [
        "#  Training a bidirectional GRU"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fs9QBZAr9ehj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras import layers\n",
        "from tensorflow.keras.optimizers import RMSprop"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UsMrScBp9zIb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# These are not Book codes. I think those codes were for last dataset jenaClimate\n",
        "\n",
        "model = Sequential()\n",
        "model.add(layers.Embedding(max_features, 32))\n",
        "model.add(layers.Bidirectional(layers.GRU(32)))\n",
        "model.add(layers.Dense(1, activation='sigmoid'))\n",
        "\n",
        "model.compile(optimizer=RMSprop(), loss='binary_crossentropy', metrics=['acc'])"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "znpmmw7ZFFTo",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 364
        },
        "outputId": "1552bf73-af9c-4896-b900-fee4c3d1771e"
      },
      "source": [
        "history = model.fit(x_train, y_train,\n",
        "                    epochs=10,\n",
        "                    batch_size=128,\n",
        "                    validation_split=0.2)"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "157/157 [==============================] - 8s 51ms/step - loss: 0.5311 - acc: 0.7258 - val_loss: 0.4315 - val_acc: 0.8016\n",
            "Epoch 2/10\n",
            "157/157 [==============================] - 7s 47ms/step - loss: 0.3130 - acc: 0.8722 - val_loss: 0.3300 - val_acc: 0.8604\n",
            "Epoch 3/10\n",
            "157/157 [==============================] - 7s 46ms/step - loss: 0.2598 - acc: 0.8996 - val_loss: 0.3100 - val_acc: 0.8844\n",
            "Epoch 4/10\n",
            "157/157 [==============================] - 7s 46ms/step - loss: 0.2237 - acc: 0.9147 - val_loss: 0.3428 - val_acc: 0.8486\n",
            "Epoch 5/10\n",
            "157/157 [==============================] - 7s 47ms/step - loss: 0.1978 - acc: 0.9258 - val_loss: 0.3114 - val_acc: 0.8734\n",
            "Epoch 6/10\n",
            "157/157 [==============================] - 7s 46ms/step - loss: 0.1742 - acc: 0.9373 - val_loss: 0.3773 - val_acc: 0.8436\n",
            "Epoch 7/10\n",
            "157/157 [==============================] - 7s 47ms/step - loss: 0.1588 - acc: 0.9427 - val_loss: 0.3278 - val_acc: 0.8804\n",
            "Epoch 8/10\n",
            "157/157 [==============================] - 7s 47ms/step - loss: 0.1438 - acc: 0.9473 - val_loss: 0.3651 - val_acc: 0.8630\n",
            "Epoch 9/10\n",
            "157/157 [==============================] - 7s 47ms/step - loss: 0.1306 - acc: 0.9549 - val_loss: 0.3356 - val_acc: 0.8848\n",
            "Epoch 10/10\n",
            "157/157 [==============================] - 7s 47ms/step - loss: 0.1240 - acc: 0.9559 - val_loss: 0.4392 - val_acc: 0.8376\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4n6wfPO-gsGI",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        },
        "outputId": "b7b85211-40cf-4d77-ba99-fbf75bff6391"
      },
      "source": [
        "evaluation = model.evaluate(x_test,y_test,verbose=2)\n",
        "print()\n",
        "print(\"Loss: \",evaluation[0]*100,\"%\")\n",
        "print(\"Accuracy: \",evaluation[1]*100,\"%\")"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "782/782 - 12s - loss: 0.5014 - acc: 0.8117\n",
            "\n",
            "Loss:  50.13570189476013 %\n",
            "Accuracy:  81.1680018901825 %\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sjMQNZNG-Hvq",
        "colab_type": "text"
      },
      "source": [
        "This performs about as well as the regular GRU layer. It’s easy to understand why: all the\n",
        "predictive capacity must come from the chronological half of the network, because the\n",
        "antichronological half is known to be severely underperforming on this task (again,\n",
        "because the recent past matters much more than the distant past in this case). "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "47Qvo1-d-Mqy",
        "colab_type": "text"
      },
      "source": [
        "#Going even further\n",
        "There are many other things you could try, in order to improve performance on the\n",
        "temperature-forecasting problem:\n",
        "\n",
        " Adjust the number of units in each recurrent layer in the stacked setup. The\n",
        "current choices are largely arbitrary and thus probably suboptimal.\n",
        "\n",
        " Adjust the learning rate used by the RMSprop optimizer.\n",
        "\n",
        " Try using LSTM layers instead of GRU layers.\n",
        "\n",
        " Try using a bigger densely connected regressor on top of the recurrent layers:\n",
        "that is, a bigger Dense layer or even a stack of Dense layers.\n",
        "\n",
        " Don’t forget to eventually run the best-performing models (in terms of validation MAE) on the test set! Otherwise, you’ll develop architectures that are overfitting to the validation set."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZA30MmDT-WTj",
        "colab_type": "text"
      },
      "source": [
        "#Wrapping up\n",
        "Here’s what you should take away from this section:\n",
        "\n",
        " As you first learned in chapter 4, when approaching a new problem, it’s good to\n",
        "first establish common-sense baselines for your metric of choice. If you don’t\n",
        "have a baseline to beat, you can’t tell whether you’re making real progress.\n",
        "\n",
        " Try simple models before expensive ones, to justify the additional expense.\n",
        "Sometimes a simple model will turn out to be your best option.\n",
        "\n",
        " When you have data where temporal ordering matters, recurrent networks are\n",
        "a great fit and easily outperform models that first flatten the temporal data.\n",
        "\n",
        " To use dropout with recurrent networks, you should use a time-constant dropout mask and recurrent dropout mask. These are built into Keras recurrent layers, so all you have to do is use the dropout and recurrent_dropout arguments\n",
        "of recurrent layers.\n",
        "\n",
        " Stacked RNNs provide more representational power than a single RNN layer.\n",
        "They’re also much more expensive and thus not always worth it. Although they\n",
        "offer clear gains on complex problems (such as machine translation), they may\n",
        "not always be relevant to smaller, simpler problems.\n",
        "\n",
        " Bidirectional RNNs, which look at a sequence both ways, are useful on naturallanguage processing problems. But they aren’t strong performers on sequence\n",
        "data where the recent past is much more informative than the beginning of the\n",
        "sequence."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FAxFQlj0krcU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}